Skip to main content

How Migrations Work

Migration is the process of controlling the database versions: thus, the below sections present the general concept of Migration and describe how it is implemented at Ucraft.

The Main Idea of Migration in General and at Ucraft​

Through Migrations, it is possible to manipulate the Database (DB), for example, create new tables or update/remove the existing ones. There can be two types of Migrations: Initial and Ongoing.

Initial Migrations are applied for creating the initial DB. Ongoing Migrations are applied to update the DB during development. Suppose that a User Table with three columns (ID, Name, and Email) is created during the Initial Migration. At a further stage of development, there occurs a need to divide the Name column into two sub-columns: First Name and Last Name. In order to achieve this, the following steps are required:

  1. To create two new columns for storing the first name and last name.
  2. To retrieve the existing name and divide it into the first name and last name.
  3. To store the results in the appropriate columns respectively.
  4. To remove the Name column.

After those changes, the User Table will contain four columns with the updated data. The given example is a basic case of Migration encountered during development.

Besides Migration, there are cases when the DB must be inserted with some static data, for example, a list of predefined email templates. If the Marketing Team wants to send customized emails to their customers, the email templates should be inserted into the DB once. There should be a process of inserting those templates into the DB for that purpose. This process is sometimes called Database Seeders or Data Fixtures. Generally, various frameworks separate the processes of Migration and DB Seeders from each other, but in the case of Ucraft, everything is implemented through Migrations, and both of the codes live in Migration.

Migration should be an idempotent action and must be run on the given DB only once: for example, if a user table is already added and the Migration is re-run, nothing should happen. The system must detect that the given Migration exists in the DB and not re-run it.
In the case of Ucraft, all the Migrations by their unique names and DateTimes are stored in a separate table. The system checks for the given Migration in the table, and if it exists, nothing happens. This system for Migration to run only once is fundamental. By default, it is possible to run a seeding for limitless times, but in the case of Ucraft, seeds are written in Migration to limit the number of runs to one, as it is in the case of Migration. For example, several system pages are generated when creating a new project in Ucraft: thus, this seed should be run only once to avoid duplicate system pages.

In the case of Ucraft, there are two types of Migrations: Strict and Eventual.
Strict Migrations are the Migrations run during the deployment for all the projects. They can work when the Migration logic does not include a project-specific action. Strict Migrations may expand the deployment time.

Eventual Migrations are the Migrations run during the project execution for every project. Eventual Migrations do not affect the deployment time. As Eventual Migrations must be run just after the deployment during the first run of the project, they can prolong the initial run of the project.

Single sign-on (SSO) will have only Strict Migrations. UC-commerce will have both types of Migrations. Builder will only have Eventual Migrations.

Several Migration cases are presented below.
If there exist one thousand projects, it is a must to manually connect to all the projects to carry out the Migration, which is impossible. Therefore, an Eventual Migration should be applied.
In the case of E-commerce, the solution of connecting to the DBs is also only sometimes effective. For example, if there is a table in the DB, and now a new column is added to it, it is practical to connect to the DB and make the change (a Strict Migration should be applied), but if there are one thousand shops and a new thing must be seeded in all the shops, again there is a need to go through all the shops and seed it manually, which takes a long time and the load is heavy to carry out. That is why the Migration should be carried out eventually. Thus, an Eventual Migration should be implemented.

The Role of Migration Dispatcher and Migrator in the System​

There are two microservices responsible for the whole process:

  1. Migration Dispatcher,
  2. Migrator.

The Migration Dispatcher starts the Migration. When it receives the message to start the Migration process for a project, it creates a new Migration with the Initial state, then it produces a message to the special Kafka Topic to start the Migration. The Migrator runs the Migration. The Migrator always listens to the Kafka Topic and reads all the messages, which include the project ID, from version, for example, 1.0.0, and to version, for example, 1.0.1. The Migrator consumes the message from the Topic and calls Accounts to set the current project mode as Under Migration. The Migrator contains the Migrations in its folder structure and runs them. If they are all successfully run, it calls the Migration Dispatcher to set the Migration Status as Success, then calls Accounts to remove the status Under Migration from the project as the project is successfully migrated and update the project version to the Latest. If something fails, the Migration status changes to Failed, and the project version remains the same.

The above-described pattern is an Orchestration-Based Epic Saga with the Migration Dispatcher as the orchestrator.

The Step-by-Step Description of the Full Flow​

  • The user visits the project: Dashboard, Visual Editor, or Public Mode.
    A request is sent to the relevant microservice depending on which of the ones mentioned above is visited. It contains a code responsible for version checks.
  • Version checks - In the Accounts DB, the project current version, for example, 1.0.0, is stored, and while deploying the product, the latest version, for example, 1.0.1, is created in a version file. While version check, those two are compared to detect the differences.
  • The Builder or E-commerce calls the Migration Dispatcher to migrate the given project.
  • The Migration Dispatcher creates a Migration in its DB mentioning the project: a unique ID for the project is generated, and the from and to versions are mentioned.
  • The Migration Dispatcher produces the current Migration to the Kafka Topic.
  • The Migrator consumes the message from the Kafka Topic.
  • Accounts is called, and the project mode is set as Under Migration.
    After this step, the project is unavailable, and the status Under Migration is displayed, not to make any changes to the project. The Migration version is set as Initial, and the project mode is set as Under Migration.
    Using the Migration ID, the users - frontend, can send a request to the Migration Dispatcher and get the information about the Migration status, which can be changed to Success or Failed from Initial. In the case of Success, the Migration is entirely implemented without any issues. In the case of Failed, something has gone wrong, and some manual interaction is required.

How to Create Migrations

If there is a task requiring Migration, regardless of the task urgency, it will be omitted from the release if the Migration for the given task is not written.
Before creating a Migration, it is necessary to find out the Migration version. After it, the user creates a new folder with the next semantic version as a name for the folder and writes the new Migration in that folder. The TimeData included in each file name allows the system to determine the order of Migrations. The file name should be descriptive and introduce what the Migration accomplishes.

💡 2022_12_01_133406_update_colors_value_to_support_dark_light_mode.php

A Migration with a specific version and name for the Builder and E-commerce can be created with the following commands respectively:

php migrator make:migration --path=migrations/database/builder/{version} {migration_name}

php migrator make:migration --path=migrations/database/commerce/{version} {migration_name}

In the appropriate folder, the Migration Blueprint is generated.

Migration Structure​

All the Migrations are native Laravel Migration classes.

A Migration class contains the up method where the Migration main logic should live.

The Migrator automatically changes the DB connection while running the Migrations and provides the project ID through the config('currentProjectId').

There are different use cases for Migrations.

An example of Migration for a DB Update (Color Settings Update)

return new class extends Migration {
    /**
     * Run the migrations.
     *
     * @return void
     */
    public function up()
    {
        $colors = DB::table('color_settings')->get(['id', 'value']);

        foreach ($colors as $color) {
            $value = json_decode($color->value, true);

            if (isset($value['color'])) {
                $newValue = [
                    'light' => $value['color'],
                    'dark'  => $value['color'],
                ];

                DB::table('color_settings')
                    ->where('id', $color->id)
                    ->update(['value' => json_encode($newValue)]);
            }
        }
    }

    /**
     * Reverse the migrations.
     *
     * @return void
     */
    public function down()
    {

    }
};

An example of Migration for a DB Update with Adding a Column

return new class extends Migration {
    /**
     * Run the migrations.
     *
     * @return void
     */
    public function up() : void
    {
        Schema::table('breakpoints', function (Blueprint $blueprint) {
            if (!Schema::hasColumn('breakpoints', 'selected')) {
                $blueprint->boolean('selected')->default(false)->after('default');
            }
        });
    }

    /**
     * Reverse the migrations.
     *
     * @return void
     */
    public function down() : void
    {

    }
};

An example of Migration for Creating a Redirections Table

return new class extends Migration {
    /**
     * Run the migrations.
     *
     * @return void
     */
    public function up()
    {
        if (!Schema::hasTable('redirections')) {
            Schema::create('redirections', function (Blueprint $table) {
                $table->id();
                $table->string('from', 255)->unique();
                $table->string('to', 255)->index();
                $table->smallInteger('type');
                $table->smallInteger('status')->index();
                $table->timestamps();
            });
        }
    }

    /**
     * Reverse the migrations.
     *
     * @return void
     */
    public function down()
    {
    }
};

There are Migration cases which are Database Seeders and need to be run only once.

An example of Migration Containing a Database Seeder

return new class extends Migration {

    public function up() : void
    {
        $pageTypes = [
            '404'                  => SystemPageType::NOT_FOUND->value,
            'Sign In Required'     => SystemPageType::SIGN_IN_REQUIRED->value,
            'Sign Up'              => SystemPageType::SIGN_UP->value,
            'My Account'           => SystemPageType::MY_ACCOUNT->value,
            'Offline'              => SystemPageType::OFFLINE->value,
            'Password Protected'   => SystemPageType::PASSWORD_PROTECTED->value,
            'Terms and Conditions' => SystemPageType::TERMS_AND_CONDITIONS->value,
            'Privacy Policy'       => SystemPageType::PRIVACY_POLICY->value,
            'Cookie Policy'        => SystemPageType::COOKIE_POLICY->value,
            'Forgot Password'      => SystemPageType::FORGOT_PASSWORD->value,
            'Reset Password'       => SystemPageType::RESET_PASSWORD->value,
            'Shipping'             => SystemPageType::SHIPPING->value,
            'Return and Refund'    => SystemPageType::RETURN_AND_REFUND->value,
            'Search'               => SystemPageType::SEARCH->value,
        ];

        $pages = DB::table('pages')
            ->where('group', '=', PageGroup::SYSTEM->value)
            ->orderBy('id')->get();

        $lastPage = $pages->last();

        $lastId = $lastPage?->id;
        $dateTime = gmdate('Y-m-d H:i:s');

        foreach ($pageTypes as $name => $type) {
            $pageModel = $pages->firstWhere('alias', '=', Str::kebab($name));

            if ($pageModel) {
                continue;
            }

            $lastId = DB::table('pages')->insertGetId([
                'previous_id'    => $lastId,
                'name_version_a' => $name,
                'alias'          => Str::kebab($name),
                'type'           => 0,
                'group'          => PageGroup::SYSTEM->value,
                'status'         => PageStatus::PUBLISHED->value,
                'params'         => json_encode(['version_a' => ['systemPageType' => $type]]),
                'created_at'     => $dateTime,
                'updated_at'     => $dateTime,
            ]);
        }

        $projectId = config('currentProjectId');

        $pages = DB::table('pages')->leftJoin('layouts', function ($join) {
            $join->on('pages.id', '=', 'layouts.renderable_id')
                ->where('layouts.renderable_type', '=', 'App\Models\Page');
        })
            ->where('group', '=', PageGroup::SYSTEM->value)
            ->select('pages.*', 'layouts.id as layoutId')->get();

        foreach ($pages as $page) {
            if ($page->layoutId !== null) {
                continue;
            }

            $structure = md5(microtime());

            DB::table('layouts')->insertGetId([
                'type'                => LayoutType::BODY->value,
                'structure_version_a' => $structure,
                'renderable_type'     => 'App\Models\Page',
                'renderable_id'       => $page->id,
                'created_at'          => $dateTime,
                'updated_at'          => $dateTime,
            ]);

            Storage::put("user_files/$projectId/layouts/$structure.json", '[]');
        }
    }
};

How to Run Migrations

With the following command, the Migrator will start to consume messages from Kafka Topic and execute the required Migrations:

php migrator go:run

📘 The command should be executed from the folder where the Migrator is installed.